Skip to content

Release engineering: Nix-built static release artifacts, published in one act - #9639

Open
cdecker wants to merge 30 commits into
masterfrom
reprobuild-nix
Open

cdecker wants to merge 30 commits into
masterfrom
reprobuild-nix

Conversation

@cdecker

@cdecker cdecker commented Oct 21, 2026

Copy link
Copy Markdown
Member

Summary

Rebuild the release pipeline so a release is produced outside GitHub and published in one act: from a tagged commit, tools/reprobuild builds the three static tarballs (amd64, arm64, armhf), the source zip, the multi-arch Docker image and the signed manifests, and publishes all of it together — artifacts and source to the download host, images to Docker Hub, the tag and the (draft) GitHub release. Nothing leads and nothing lags: a user can upgrade the moment they learn there is something to upgrade to.

The build is a Nix derivation — the same in-tree derivation nix build produces, as a static/cross variant — so "reproducible" is enforced by data and guards, not by hope:

nix/
├── release-rows           # 3 rows + the certified nixpkgs rev; read by flake and driver, no tooling
├── pkgs/static-pie.nix    # the shared half-recipe: static-pie link, mimalloc, path remap
├── pkgs/rust-bin.nix      # release Rust toolchain from upstream dist, pinned via signed channel
├── release-manifests/     # per-row inputs incl. recipe content-hashes -> a diff to a reviewer
└── ...
tools/reprobuild           # the only host entry point (docker + coreutils + python3; no host Nix)
contrib/reprobuild/        # the checks: check.sh, smoke.sh, portability.sh, two-tree-probe.sh
certify   re-pin nixpkgs + manifests, build from source, run the acceptance suite   (deliberately not in `all`)
all       zip + build (3 rows, parallel) + fetch + docker + sign
sign      SHA256SUMS-<ver> (+ a separate -armhf manifest), 3 signatures required to publish
verify    co-signer: rebuild and reproduce every file in the manifests from the commit
publish   upload -> read back + check against the signed manifest -> images -> tag -> GitHub release

Two guards make the toolchain an auditable input: the certified rev in nix/release-rows must equal flake.lock (nix flake check trips otherwise), and every certify re-generates the per-row input manifests, whose recipe hashes make an edit to the linking recipe show up as a manifest diff.

What goes away: the per-distribution builder images (Ubuntu/Fedora/Alpine route), their pin lists and hooks, and the GitHub Actions release/image workflows. release.yml keeps PyPI only; the driver refuses to push a tag while that workflow would race it. Native builds are unaffected except that a configure probe fix (a probe that always answered "no" for lack of a main()) re-enables -ffunction-sections/--gc-sections, i.e. slightly smaller native binaries.

The VLS signer ships in no release here: CLN refuses a signer without splice support and no released VLS has it, so the binaries could only ship unable to start. The derivations and harnesses stay; bringing it back is a pin bump, a green regtest, one flag.

Evidence

  • Artifacts: all builds three tarballs that pass the ELF gate — 40 ELFs per row, all static-pie, --strip-all, build-id present — and every ELF boots under pinned qemu on the oldest supported Pi CPUs (cortex-a53, cortex-a7): 0 signal deaths.
  • Self-reproduction: nix build --rebuild reproduces all three rows from one tree; cross-machine byte-equality (two machines) was established on the previous toolchain and is being re-run against this tree's toolchain bump (gcc 16.2 / rustc 1.98) as we speak; the armhf Rust rows are carried in a separate manifest pending that divergence, which never blocks a release.
  • Portability: the static tarballs run unmodified on Ubuntu 22.04/24.04/26.04 and Fedora 43 across all architectures (11/11 under the previous toolchain; the matrix is a driver verb and is re-run on every toolchain move).
  • Docker: the multi-arch image consumes the signed tarballs and runs lightningd --version on all three platforms under binfmt.

Merge Danger

Door: two-way until the first publish through it — no tag moves and nothing publishes from CI by merging this; rollback is re-enabling the old workflows. The one-way door is the first release made this way: the reproducibility promise those tarballs make is forever.

Blast Radius: release tooling, release/publish CI, the runtime Dockerfile, docs and configure. The developer path (./configure && make, make install to /usr/local) and packaging are untouched; the configure probe fix changes native binaries' size/sections, not semantics.

@cdecker
cdecker force-pushed the reprobuild-nix branch 2 times, most recently from 80cd2ad to 8f6ccc7 Compare October 22, 2026 10:20
@daywalker90

Copy link
Copy Markdown
Collaborator

@ 8f6ccc7

./tools/reprobuild all --mtime 2026-10-23
cat release/SHA256SUMS-v26.06-618-g8f6ccc7
# Core Lightning v26.06-618-g8f6ccc7
# rev 8f6ccc73c5dd0ac8c205cc4855de5a1d6362de30
# narHash sha256-NPSRLvqpwyxjBcB8OHeXRc8OgNW1tcAMd3A2Z9HeRKI=
# submodules true
# mtime 2026-10-23
# nixpkgs-certified b4fd65b198c599cbe814fcb9f42d25d021595ec9
29d2e7025123e07cf5cc739e2671d7e8294ea08cef6089e3ff8358c25da44b38  clightning-v26.06-618-g8f6ccc7-static-amd64.tar.xz
0845a7f72a11adc14b0e15694ee823ac2ca02609264de17ba880b7ac11567b04  clightning-v26.06-618-g8f6ccc7-static-arm64.tar.xz
fe73d9553c8841b9708252bd3ecb4692f3585fe50f0a75d0d71a06e870c18dde  clightning-v26.06-618-g8f6ccc7.zip
cat release/SHA256SUMS-v26.06-618-g8f6ccc7-armhf
# Core Lightning v26.06-618-g8f6ccc7
# rev 8f6ccc73c5dd0ac8c205cc4855de5a1d6362de30
# narHash sha256-NPSRLvqpwyxjBcB8OHeXRc8OgNW1tcAMd3A2Z9HeRKI=
# submodules true
# mtime 2026-10-23
# nixpkgs-certified b4fd65b198c599cbe814fcb9f42d25d021595ec9
f7ecaa2a55c67c2e156484731457a967832a618460c279317e01428b7d59b55d  clightning-v26.06-618-g8f6ccc7-static-armhf.tar.xz

I still think we should not have the separate armhf shasum file since everyone can cross compile it now.

@cdecker

cdecker commented Oct 23, 2026

Copy link
Copy Markdown
Member Author

Agreed, and I am working onusing the head commits timestamp as the mtime so we have one fewer potential mismatches, and it's reusing a datetime we already agree on anyway.

@cdecker

cdecker commented Oct 25, 2026

Copy link
Copy Markdown
Member Author

Cross-machine reproduction: the armhf divergence is gone, and what remained was the calendar.

@daywalker90 I reproduced your run exactly — dropped your two manifest files into release/ and ran

./tools/reprobuild verify release/SHA256SUMS-v26.06-618-g8f6ccc7
./tools/reprobuild verify release/SHA256SUMS-v26.06-618-g8f6ccc7-armhf

verify pins rev and mtime from the manifest headers (# mtime 2026-10-23), and all four files reproduced byte-for-byte, armhf f7ecaa2a… included.

That run shape explains the earlier "armhf differs across machines" data points: at 8f6ccc73c, our two Linux build hosts (AMD and Intel) produced identical armhf hashes when they shared a build day (2026-10-22) — but your --mtime 2026-10-23 build legitimately produced different bytes from theirs. The rustc-1.89-era divergence (the reordered libunwind ARM-EHABI string table) is gone under the current toolchain; what was left was the date, not codegen.

Which is exactly what the mtime commit on this branch removes as an input: --mtime now defaults to the built commit's UTC date, so the build day can no longer move the bytes. Measured at bfa477bb3, one commit past your run:

artifact host A (built 2026-10-24) host B (same day) host A rerun (built 2026-10-25)
static-amd64 7db078dc35c35765… same ✅ same ✅
static-arm64 0432ec61839115ce… same ✅ same ✅
static-armhf 0a7440bae4af1b24… same ✅ same ✅
source zip ba922977abe3a229… same ✅ same ✅

The day-rollover rerun's manifest header says # mtime 2026-10-24 — the commit's date — even though the build ran on the 25th. (Full hashes from these runs are in the manifests; happy to paste them verbatim.)

One more machine is queued: an Apple-Silicon Mac, where the run currently dies in Docker Desktop's virtiofs layer refusing the chmod a Nix store needs — environment plumbing on our side, nothing about the build itself.

Consequence, as you suggest: the separate SHA256SUMS-<ver>-armhf can go. armhf no longer needs scoping out of the required co-signed set — one manifest, three signatures, no asterisk. The follow-up commit here would retire the split-manifest writer, verify's withheld-signature path for -armhf, and the docs' caveat; older releases keep verifying fine, since verify reads whatever manifest it is handed.

@madelinevibes madelinevibes added this to the v26.12 milestone Oct 26, 2026
@cdecker

cdecker commented Oct 26, 2026 •

Copy link
Copy Markdown
Member Author

Edit (2026-10-26): the fetch-noise suppression described in the first half of this comment was pulled off the branch — the shim was too much machinery for the noise it hides. The branch head is back at 2da593278 (v26.06-620-g2da5932), whose bytes are the canonical set for verification. The --host-nix equivalence result below stands as an observation — both routes produced bit-identical output from the same tree — but the 621 hashes quoted belong to a commit that is no longer on the branch.


Two follow-ups on the build hygiene front, both landing on top of the merged-manifest state.

The build log got quiet. nix resolves the flake with submodules=1 by running, per submodule, git ls-remote and git -C <cache> fetch --progress --force -- <url> refs/* — a catch-all refspec with forced progress — so git printed one status line per ref (tens of thousands of PR refs as [new ref], plus byte counters) on every driver invocation, because the container's /root/.cache/nix died with each --rm run. Modern git removed the fetch.output config, so the only knob left is the --quiet flag nix does not pass: 5ff6b25 mounts a small git shim into the container that inserts it (after the option flags, so it beats --progress; before the positionals, where it would read as a refspec), and persists that fetch cache beside the store so the ref crawl happens once per machine instead of once per call. Errors still print, and the cache persistence moves no output bytes — cold and warm runs of the same commit hash identically.

--host-nix produces the identical artifacts. The driver has an opt-in route for machines with their own sandboxed nix, so I tested whether it agrees with the pinned-container route: same machine, same commit (5ff6b250f → v26.06-621-g5ff6b25, default mtime), built both ways, one after the other:

artifact container (nix 2.35.2 image) host nix (2.35.3) cmp
clightning-v26.06-621-g5ff6b25.zip identical identical ✔
…-static-amd64.tar.xz identical identical ✔
…-static-arm64.tar.xz identical identical ✔
…-static-armhf.tar.xz identical identical ✔

Full hashes for anyone cross-checking this head:

399817c3b346fe1b094fdc0892e6f01df234a42ecff7e5908cf84bafc20d4490  clightning-v26.06-621-g5ff6b25.zip
27471a49a7a1a9cc4484c5eefa3961ec9941db76b97526d059954f815a57b922  clightning-v26.06-621-g5ff6b25-static-amd64.tar.xz
57ea33beaa9dbf025bff8e5dfec34fae1ae100e7c85d2667c84a1e5b92abf351  clightning-v26.06-621-g5ff6b25-static-arm64.tar.xz
98ab8c6dffce40a8b85a550d6c209b5239a55927dd32f3e69e4d1c01962ccf48  clightning-v26.06-621-g5ff6b25-static-armhf.tar.xz

Incidentally this doubles as a cross-version check — the two routes ran different nix minor versions (2.35.2 in the pinned image vs 2.35.3 on the host) and the compiled outputs, armhf included, are byte-for-byte the same. The container pin is belt-and-braces, not a load-bearing difference. The host route was silent too: the daemon's fetch cache persists on its own, so the once-per-machine crawl already covers it.

CI on 5ff6b250f: 45/45 green (one ASan test_gossip_seeker_autoconnect flake on the first attempt passed on rerun; unrelated to this tooling).

@cdecker

cdecker commented Oct 26, 2026

Copy link
Copy Markdown
Member Author

Closing the loop on the fetch-noise experiment: the suppression was pulled off the branch (5ff6b250f is gone; head is 2da593278 again).

What it took to work: a git shim mounted into the container ahead of PATH, doing argv surgery — --quiet has to land after the subcommand's option flags (git has no global -q, and a later --progress would override an earlier --quiet), before the positionals (ls-remote reads anything there as a ref name), while keeping git -C <dir> --git-dir .-style global options intact — plus an image-specific GIT_REAL path and a second mount for the persistent fetch cache. Every layer was load-bearing and silent when it broke: a wrong exec path or flag position turns nix's submodule fetching into an error or, worse, a silently different fetch. That is too much machinery, in the release path, for the privilege of not seeing a thousand [new ref] lines on a machine's first fetch.

The finding worth keeping is in the comment above: the container route and --host-nix (different nix minor versions) build bit-identical artifacts, so the pinned image is belt-and-braces, not load-bearing. The noise itself is transport chatter around content-addressed inputs — ugly, not wrong. The driver stays as it is.

@daywalker90

Copy link
Copy Markdown
Collaborator
cat release/SHA256SUMS-v26.06-620-g2da5932
# Core Lightning v26.06-620-g2da5932
# rev 2da59327866264e164da00ddb016036ed35ce4f2
# narHash sha256-6umz898VLAay/jBqB5RMnnlLjyY/1aBYJ4hEwfU9Y7s=
# submodules true
# mtime 2026-10-25
# nixpkgs-certified b4fd65b198c599cbe814fcb9f42d25d021595ec9
3246e96446d6eab9c498a854124f755a6dc44fec57f8007df49a36d24eec89d1  clightning-v26.06-620-g2da5932-static-amd64.tar.xz
6bcd950b82cd8dec2d724bbccf9a889edd61085ec76b9f91f39155a9d0dba469  clightning-v26.06-620-g2da5932-static-arm64.tar.xz
cfe2b9216a84fc8e2f11d41fb1c02db3eec2cb6425a380d41660e2db36952b0f  clightning-v26.06-620-g2da5932-static-armhf.tar.xz
fa74010fa7a106a8669992ae70fc9f41fc7c36fd9c645473265a9d9e51e6059e  clightning-v26.06-620-g2da5932.zip

@madelinevibes madelinevibes removed this from the v26.12 milestone Nov 9, 2026
@cdecker

cdecker commented Nov 18, 2026

Copy link
Copy Markdown
Member Author

Thanks @daywalker90 and sorry for not following up earlier. I was able to verify your SHA256SUMS file without issue, on both Linux (Ubuntu 26.04) and MacOS (via lima). I'll rebase this PR and mark it as undrafted, and then push another snapshot for others to verify.

cdecker added 18 commits November 18, 2026 13:43
Three changes a static or cross build needs, none of which affects a native
dynamic one:

 - the configurator itself is compiled with CONFIGURATOR_CC.  It runs on the
   build machine, so when cross-compiling it must not be built for the host;
   its test programs are still compiled with CC and are now run through
   CONFIGURATOR_WRAPPER (e.g. qemu-aarch64), which makes the five places the
   build runs its own output explicit instead of relying on the kernel's
   binfmt registration.
 - POSTGRES_LDLIBS and POSTGRES_INCLUDE can come from the environment. A
   static libpq.a needs archives pg_config does not report (-lpgcommon
   -lpgport -lssl -lcrypto), and the no-pg_config branch used to *assign*
   these empty, discarding what the caller set -- which silently produced a
   lightningd with no PostgreSQL support at all.
have_function_sections() links its test program with -o, so without a
main() the link dies on "undefined reference to `main'" -- on every
platform, not just a fully static one against musl's crt1.o.  The probe
therefore answered "no" however capable the toolchain was, and
HAVE_FUNCTION_SECTIONS has been 0 everywhere: -ffunction-sections and
-Wl,--gc-sections were silently off for every build this project shipped.

Adding the main() turns them on wherever the toolchain supports them,
which is a change to native builds too: smaller binaries through
section gc, and nothing else -- the flags do not alter semantics, and
warnings are unaffected.
The configurator's loop sets every variable config.vars mentions, so
afterwards a stored empty value is indistinguishable from an environment
setting, and `: ${POSTGRES_LDLIBS=...}` skipped the pg_config probe.  A
--reconfigure whose stored value was empty therefore built a lightningd
with no PostgreSQL support at all, without a word about it.

Whether the value came from the environment is now recorded before
config.vars is read; anything else is probed again, as a first run would.
The build runs some of its own output, which a cross build cannot do:

 - cdump-enumstr is compiled with CC_FOR_BUILD and taken out of
   ALL_PROGRAMS: it is run to generate source, so it must be a build-machine
   program, while libccan.a is host code.
 - tools/headerversions and plugins/sql --print-docs run through
   CONFIGURATOR_WRAPPER. The sql one had its failure masked by a pipe to jq,
   which shipped a lightning-sql.7 with no TABLES section.
 - external/Makefile passes BUILD/MAKE_HOST so libwally and libbacktrace
   configure as cross rather than native, and libbacktrace's generated
   backtrace-supported.h is declared as a dependency of the objects -- a -j
   race that only showed up where gcc did not happen to provide its own copy.
 - cln-rpc honours RUST_TARGET_DIR so the Rust artifacts land per target.

That last dependency is an *order-only* prerequisite, and the header is
deliberately not a member of EXTERNAL_HEADERS, which is subtler than it looks.
EXTERNAL_HEADERS depends on the phony submodcheck, so every member of it is
permanently out of date; that is harmless for headers we merely check out,
because a target with no recipe and no missing file is not propagated as
updated. A generated header behaves differently at both ends of that rule: as
a member with no recipe it is a nonexistent file, which make imagines to have
been updated and so invalidates every object; as a member with a recipe it
re-runs libbacktrace's configure on every single make. Order-only says what is
actually meant -- it must exist before anything compiles, and its timestamp is
not a reason to recompile.

A grouped target then says the archive and the header come from one recipe,
and the header joins TESTPACK_EXTRAS for the same reason $(EXTERNAL_LIBS)
already had to: the testpack prunes external/, so otherwise it never reaches
the phase that only checks, and make would build it there with a compiler
stubbed out to refuse.
…ation

The release artifacts come from the same derivation as packages.default,
evaluated under pkgsStatic (and pkgsCross for the arm rows), rather than from
a second release-only expression: the thing tested hardest each release is
then the thing Nix users install.

release = true switches on what the reprobuild contract needs and nothing the
ordinary build wants: version and mtime from the driver, one `make install`
(install's own prerequisite graph already covers everything shipped), an
offline cargo vendor dir, --prefix=/usr into a DESTDIR, strip --strip-all with
an in-derivation assertion that no shipped ELF keeps .symtab/.debug_* or loses
its build-id, and dontPatchELF so fixupPhase cannot rewrite signed bytes.

static-pie.nix carries the link recipe, including two gaps nixpkgs does not
cover: rustc still links crt-static musl targets non-PIE and self-contained,
so every row rewrites its link line; and gcc cannot static-pie on armv7 at
all, so that row links through an explicit-crt wrapper. Both are needed to
keep ASLR on every shipped executable.

Three changes stop the release bytes tracking the source tree: a constant
unpacked source directory, a remap for the cargo vendor dir, and a pinned
NIX_OUTPATH_USED_AS_RANDOM_SEED -- without the last one, nixpkgs seeds every
C compile from $out and --build-id=sha1 hashes the unstripped output, so a
tracked file that is never compiled moved every build-id.
…y consume

nix/release-rows is the matrix as data: arch, package-set path, Docker
platform. The flake generates cln-release-static-<arch> and
vls-release-static-<arch> from it and the driver reads it with a plain
`while read`, so there is one source of truth and no tooling on either side.

Two guards, both only on the release path:

 - the rows record the nixpkgs revision the release is certified against.
   A `nix flake check` entry compares it with flake.lock and the release
   derivations refuse to evaluate if they differ, so a contributor's
   `nix flake update` cannot move gcc, rustc and openssl under three signed
   tarballs without a word. The error names the three ways out.
 - each row commits a short input manifest -- direct inputs with versions,
   toolchain versions, build flags, the release environment -- checked at
   eval. A distribution-motivated change that reaches the release then shows
   up as a diff a reviewer who does not read Nix can read.
Every signed artifact is a Nix output, so `nix build --rebuild` covers the
bytes that ship. The zip was the last one assembled on the host, out of
`git ls-files` of the *working tree* -- which is not the commit a manifest
names, so a co-signer standing on a different branch rebuilt a different zip.

vls.nix builds VLS's remote_hsmd_socket over the same rows. It is not
rustPlatform.buildRustPackage: that would resolve rustPlatform to the row's
toolchain and compile a musl rustc from source inside the inputs of a signed
artifact. The crate closure is one fixed-output vendor derivation, and a git
shim answers the one `git describe` build.rs asks for, since a fixed-output
source fetch has no .git and every shipped binary would otherwise report
git-desc-failed.
The manifests describe the build's inputs -- which package versions, which
flags -- and said nothing about the files that decide what is done with them.
So an edit to static-pie.nix could change how every shipped binary is linked,
move the output bytes, and leave the manifest byte-identical: guard 2 would not
fire, and the promise that nothing reaches a signed release without a
reviewable diff did not hold where the most consequential edits happen.

It is not an oversight in the flags section but a structural gap. The compiler
wrapper reaches the build through configureFlagsArray inside the build phase
and the Rust linker through an exported variable; neither is in the
configureFlags attribute the flags section is generated from, and neither could
be without duplicating the recipe into its own description.

So name the recipe by content hash instead: default.nix, static-pie.nix and
flake-module.nix for a CLN row, vls.nix and static-pie.nix for a VLS one. Any
edit to them now moves the manifest, trips the guard, and has to go through
certify like a nixpkgs bump.
Replaces build-release.sh and repro-build.sh. The host needs docker,
coreutils and python3 (3.10+): every nix invocation runs inside a
digest-pinned nixos/nix container against a store under XDG_CACHE_HOME, so no
host nix.conf -- sandbox, substituters, filter-syscalls -- is an input to a
signed artifact. --host-nix is an opt-in for whoever has Nix, and both routes
produce identical bytes.

Python rather than shell, importing only the standard library and run through
plain python3 rather than uv: a co-signer reproducing a release should need an
interpreter, not a package manager. The verbs that emit signed bytes were held
to that -- the seven release derivations instantiate to the same .drv paths as
the shell driver produced, and both manifests come out byte-identical -- so
the rewrite is a change of language, not of output. It also drops the
hand-rolled OCI JSON parsing the shell version needed to avoid a jq
dependency; `json` is in the standard library.

Verbs: fetch, build, vls, zip, docker, sign, verify, certify, publish,
disclose, accept, all. `all` stops at sign; nothing pushes without publish or
disclose, and certify is deliberately outside `all` so a release day cannot
quietly re-certify a toolchain nobody examined.

What the driver refuses to do is most of its value:

 - build a release from a dirty tree: nix caches the worktree it fetched, so
   the artifacts could be labelled with a commit they were not built from.
 - build unsandboxed. Under default docker nix cannot set up the sandbox and,
   with sandbox-fallback at its default, builds without one *silently* -- the
   build directory then ends up baked into every locally compiled Rust
   artifact, and no such build reproduces any other.
 - sign a manifest whose files came from different trees, or publish
   artifacts that did not come from the tag being published.
 - start a build that would end in a signature it cannot make: the signing
   key is exercised on a throwaway file first.

Manifests follow decisions 19 and 45: SHA256SUMS-<ver> for what a co-signer
is expected to reproduce and SHA256SUMS-<ver>-armhf for the row that does not
reproduce across machines, each with a header recording rev, narHash,
submodules, mtime and the certified nixpkgs revision -- so a hash is never
reported without the tree it came from. verify takes the rev and the mtime
from the manifest it is handed, not from the verifier's checkout.
check.sh asserts the strip policy on the shipped bytes -- stripped,
static-pie, build-id present, no embedded library search path -- and is a
precondition of sign, not advice. portability.sh runs the distro matrix
(three Ubuntus and Fedora) with smoke.sh inside each container: --version, a
regtest start and a getmanifest handshake with all 27 plugins. `all` runs the
newest Ubuntu only; the full matrix belongs to the acceptance suite.

two-tree-probe.sh builds one row from two trees differing by a single
tracked, unshipped, uncompiled file and requires identical bytes. It replaces
a "zero /nix/store references" invariant, which was both unreachable
(lightningd keeps OpenSSL's store paths deliberately) and insufficient: a
variant whose bytes tracked the tree carried no offending string at all. It
runs per row, because the defect it last caught was armhf-only.

vls-regtest.sh runs the CLN-VLS pair on regtest against a test-only vlsd,
since remote_hsmd_socket only listens for a signer to connect and proves
nothing alone. fetch-bitcoin.sh downloads bitcoin-cli and counts good
signatures from the vendored keys rather than trusting gpg's exit status.

pins carries the container digests and the Bitcoin Core pin; publish.vars the
public bucket, its endpoint and the prefix the source archive is served from.
The image consumes the static tarball for its platform instead of compiling
anything: debian:trixie-slim pinned by index digest, the tarball entering
through a mount so it is never stored in a layer, bitcoin-cli fetched and
GPG-verified on the host beforehand.

The -vls overlay also gets the environment its signer cannot start without.
remote_hsmd_socket reads BITCOIND_RPC_URL, VLS_NETWORK and VLS_CLN_VERSION and
panics if any is missing, and lightningd then waits forever on an hsmd that
never answers, with no error of its own. The network follows lightningd's; the
bitcoind URL cannot be guessed, so it is refused with an explanation instead.
The Ubuntu and Fedora builder images, the Alpine/apk static builder,
their pin lists and hooks, the VLS pseudo-distro and both old driver
scripts all go: the matrix is one static build per architecture now,
produced by a Nix derivation, so there is no image of ours to build and
no per-distribution directory to keep in step.

Keeping them as a fallback was considered and rejected -- they produce
Ubuntu tarballs, a matrix that no longer exists, so invoking them would
ship a different set of artifacts than the one promised. Any historical
release is reproduced from its own tag, where these files still are.

Leftovers from the same route, all found in review:

 - contrib/cl-repro.sh built the cl-repro-jammy/noble/resolute builder
   images from Dockerfiles that no longer exist. Nothing referenced it.
 - rust-toolchain.toml only ever affected contributors: its own header
   says it is not what the release is built with, since the toolchain
   comes from the certified nixpkgs revision. What it did do was install
   a 1.85 toolchain plus six cross targets nobody building CLN needs,
   under the `minimal` profile, so a rustup user lost rustfmt and never
   had rust-analyzer. A pin that helps nothing and degrades every
   contributor's setup is worse than no pin.
 - .dockerignore still explained itself in terms of builder images that
   COPY their pins and rust-toolchain.toml, and kept both visible for
   them. The runtime Dockerfile's only main-context input is the
   entrypoint; the release tarball, bitcoin-cli and VLS all arrive as
   named build contexts.
 - release.yml had a `gate` job whose entire purpose was to join several
   parallel builds. One build is left, so it joined nothing.

Changelog-Changed: release binaries are now one static build per architecture (amd64, arm64, armhf) instead of non-static builds per distribution (Ubuntu, Fedora).
The release binaries and the images are produced and signed locally by
tools/reprobuild, from one static build per architecture, and published from
the release captain's machine. So release-build.yml and release-publish.yml go
with the scripts they called, repro.yml with the builder images it rebuilt
nightly, and docker-build.yml/docker-publish.yml with the CI image path --
docker-publish was also the job that pushed by digest and applied `latest` to
any non-rc tag, neither of which a release under embargo can afford.

"Release 🚀" keeps the tag check and PyPI. tools/reprobuild disclose refuses to
push a tag while it is enabled, because a tag push would publish the pyln
modules on its own schedule rather than the captain's.
The reproducible-builds page loses the per-distribution builder images and
gains the driver: what the artifacts are, the two guards on what the build
consumes, per-release manifest names with the identity header, and co-signing
as a single `verify` that takes the commit and the timestamp from the manifest
it was handed -- including how to fetch that commit when it is only on the
private mirror.

The release checklist becomes one sequence for rc, final and point releases:
all, three signatures, publish at T0, disclose when the source may go out. The
docker-images page points at the driver instead of the builder images.

Both note what the release does not currently contain: no VLS signer binary
and no -vls image, because Core Lightning offers channel splicing by default
and refuses a signer that cannot sign splices, which no released VLS does yet.
…t up front

`all` preflights the signing key before it builds anything, so that an hour of
building cannot end on a key nobody can reach. For a key on a smartcard that
has a cost the preflight was not written for: it means being at the machine at
the start, an hour before there is anything to sign, and again at the end.

--wait-for-key inverts that trade. `all` skips the up-front check, builds
unattended, and waits when it reaches sign -- 30 minutes by default. Absence
stops being an error and becomes something to wait for; what is given up, and
only under the flag, is failing early on a key that is broken rather than
merely absent.

It polls the key rather than reading a keypress. Runs this long are started
detached, where there is no stdin to read from but there is a log to tail, and
a poll answers "is the key ready" where a keypress only claims it is. Each poll
is a real signature of a throwaway file, which is also what unlocks a
smartcard, so success is proof rather than inference.

`sign` now checks the tree before the key: the tree check is instant, and
summoning someone to unlock a smartcard only to refuse the run for a dirty tree
is the wrong order.
The release derivation used --prefix=/usr, so the tarballs unpacked over `/`
and the docs had to tell people to do that. A release tarball is not the place
to move a project's install directory: every earlier release put its binaries
in /usr/local, `make install` still does, and someone upgrading would have
ended up with two copies in two prefixes.

So back to --prefix=/usr/local, and the documented command is the one it always
was, `-C /usr/local --strip-components=2`.

The harnesses stop hardcoding the prefix rather than swapping one spelling for
another: check.sh and vls-regtest.sh locate lightningd in the extracted tree,
and smoke.sh takes it from PATH and derives the plugin directory from it. Which
prefix the release installs under is the derivation's business, and three
scripts that each had to know it were three things to keep in step.
Binaries and source now go out together, so the two-verb split goes: `publish`
uploads the artifacts and the source zip, reads them back and checks them
against the signed manifest, pushes the images, pushes the tag and creates the
GitHub release. Least-reversible last, and nothing left that can half-publish a
release. The runbook loses the T0 / T0+14 structure with it.

Stripping stays, re-justified: it was argued for as a security property that
only held while source lagged binaries. The real reasons are that it keeps the
tarballs small, matches every earlier release, and costs nothing because the
build-id survives for symbolising crashes against the unstripped twin.

The comments are rewritten throughout to stand on their own. They referred to
ticket numbers and decision dates that a reader of this diff has no access to,
and an earlier pass at removing them had left duplicated sentences
(default.nix's header said the same thing twice) and clauses that stopped
mid-thought ("rustc's static-pie gap, which is"). static-pie.nix now opens by
saying what it is -- the shared half of one recipe, imported by both CLN and
VLS, holding workarounds for two upstream gaps -- which is the question it kept
being asked.

publish.vars loses a provisioning narrative that named an internal ticket, the
cloud project, IAM role details and the password-manager vault and item. None
of it is needed to run the release, and a public repository is the wrong place
for it.

Two bugs fell out of writing the above:

 - publish_vars() and pins() matched keys with `[A-Z_]+`, which does not match
   a digit, so REPRO_S3_ENDPOINT, REPRO_S3_BUCKET and REPRO_S3_PROVIDER were
   silently absent and `publish` would have died on a KeyError at the upload.
   The shell driver looked each key up by name and did not have this.
 - `verify`'s advice on a missing tag pointed at a private mirror that is not
   part of this flow.
One more review step survives the collapse: the GitHub release page is
created as a draft, so the captain has the hand check the flow has always
had before anyone reads it. Everything else is public by then.
The flake inputs were 11 to 15 months old; all six move to today. nixpkgs goes
from 2025-10-25 to 2026-09-29, which carries the toolchain with it:

  gcc     14.3.0 -> 16.2.0
  rustc   1.89.0 -> 1.98.1
  python  3.13.8 -> 3.14.7

The container pins move too. The Nix builder goes to 2.35.2, still by index
digest. skopeo goes to v1.22.3 rather than the v1.24.1 that exists as a GitHub
release: quay.io/skopeo/stable does not publish 1.24.x, and the container track
is what this pin names.

One thing the bump broke, and it is the guard working: the 32-bit ARM row
carried `hardeningDisable = [ "pie" ]` on libpq, because postgres could not
configure there with nixpkgs' pie hardening and pkgsStatic's -static together.
`pie` is no longer one of nixpkgs' hardening flags, so naming it now aborts
evaluation rather than being ignored. Dropped rather than translated to a
neighbouring flag that is not obviously the same thing; whether the original
problem is back is a question only an armhf build answers, and that build is
still running.

Guard 1 refused the first build after the bump, before anything was compiled,
which is exactly what it is for: the certified revision in nix/release-rows no
longer matched flake.lock. Re-pinned and all six manifests regenerated through
`certify --no-verify`.

The bump to current nixpkgs failed to evaluate with "expected a set but found a
function", always on some package's `src`. The cause was our own overlay: it
replaced `fetchurl` with a plain lambda, and `fetchurl` is no longer a bare
function. It is a set carrying __functor, override, extendDrvArgs and friends, so
flattening it broke every nixpkgs call site that overrides it -- which is most
`src` attributes, via packages we never name.

The overlay existed because crates.io used to reject a User-Agent starting with
"curl/". Removed rather than rewritten against the new interface, because it is
no longer needed: a full amd64 release build fetches everything it needs without
it. If that regresses, the way back is `extendDrvArgs`, not another lambda.

flake.nix joins the recipe hashes in the input manifests while we are here. Its
overlays reach every package in every row, so an edit there can move the shipped
bytes -- which is not hypothetical, since this is exactly what the fetchurl
overlay did -- and until now guard 2 could not see it.

GNU autogen goes too, as an unused build input rather than as part of the fix: it
was the *first* thing to fail (through guile 2.2.7 and its boehm-gc, broken under
pkgsStatic), but removing it only moved the failure along. Nothing in this build
invokes autogen -- libwally's tools/autogen.sh is a shell script that runs
autoreconf -- so it was dragging a compiler and a garbage collector into the
closure for nothing.

Verified on the amd64 row: builds clean under gcc 16.2.0, rustc 1.98.1 and
python 3.14.7; 40 ELFs all static-pie, stripped, with build-id; the tarball
installs under /usr/local; and the portability smoke test passes on ubuntu:26.04
with 27 of 27 plugins answering getmanifest.
cdecker added 12 commits November 18, 2026 13:44
CLN's configure defaults COPTFLAGS to -Og, which is gcc's "optimise without
getting in the debugger's way" -- roughly -O1. Nothing overrode it, so every C
file in every release this project has shipped was compiled that way; the Rust
half was fine, since it builds with RUST_PROFILE=release.

Measured before changing it, three interleaved pairs of runs on one tree
differing only in this flag, medians of three:

  rpc_listpays_c8          +17.0%      ranges separate
  grpc_getinfo_c8          +17.0%      ranges separate
  grpc_listpeers_c1         +8.1%      ranges separate
  rpc_listinvoices_b_c8     +6.7%      ranges separate
  grpc_listpeers_c8         -3.3%      ranges separate

Eight further phases were inconclusive -- their per-run ranges overlap -- and
the clnrest ones varied 2.75x within a single side, so they say nothing at this
sample size. The wins are on list operations, which is where sustained C-side
iteration and serialisation happens, and that is the shape one would predict.
The one separated regression is small and unexplained; it is not worth 17% twice.

RSS is unchanged: lightningd 53028 vs 52960 kB, every plugin within noise. So
this is not a size-for-speed trade.

Warnings are unaffected. default_cwarnflags only drops -Wno-maybe-uninitialized
at -O3, and -Werror is added for debug builds only, so -O2 changes neither.

Verified on the built artifact: 40 ELFs all static-pie, stripped, with build-id,
and 27 of 27 plugins answering getmanifest on ubuntu:26.04.

This moves the shipped bytes, so the reproduction and portability
measurements are retaken against it, together with the toolchain bump.
The release used `pkgs.buildPackages.{rustc,cargo}` from each row's
package set.  Hydra does not build `pkgsCross.<t>.pkgsStatic`, so on
the arm64 and armhf rows cache.nixos.org had no rustc and every build
compiled rustc from source first (~4.5 min on 24 threads, per row, per
fresh store, and again for every co-signer).  It is still true at the
current nixpkgs pin: the 1.98.1 static cross rustc is not cached for
either row, while the plain cross one is.

nix/pkgs/rust-bin.nix instead fetches rustc, cargo and rust-std from
static.rust-lang.org, pinned by the sha256s in the signed
channel-rust-1.98.1.toml -- the same version nixpkgs carries.  One host
toolchain serves all three rows and nothing Rust-related is compiled.
nixpkgs bootstraps its own rustc from the same binaries, so this adds no
trust the release did not already extend.

The musl objects rustc links self-contained (libc.a, crt1.o, rcrt1.o,
...) are replaced with the row's own musl, as nixpkgs' static rustc
does, so the C and Rust code link one libc.  crtbegin/crtend and
libunwind stay upstream's.

rust-bin.nix joins the recipe hashes, and the input manifests record
each component's sha256.  This moves the release bytes, but only the
Rust half: built with the same version and mtime, the amd64 tarball's
33 C binaries are byte-identical to the parent commit's and only the 7
Rust plugins differ.  Verified on all three rows: check.sh passes
(static-pie, stripped, build-id, same ARM attributes), portability.sh on
ubuntu:26.04 passes 27/27, and `nix build --rebuild` reproduces all
three.
…argo run

`tools/reprobuild all` built each artifact with its own `nix build`: the
zip, then amd64, then arm64, then armhf, then --rebuild each of them the
same way.  The rows do not depend on each other, so nix could overlap
them, but was never asked to.

The driver now puts every artifact a verb needs into one attribute set
and builds it with one `nix build`, reading the output paths back in
order.  `all` and `verify` build the zip, the rows and (when it ships)
VLS together; `build`, `vls` and `zip` still work alone.  --rebuild
checks all of them in one run too.

nix's default `cores = 0` gives every derivation all CPUs, so three rows
at once would each run make -j$(nproc).  The driver passes
--max-jobs <rows+1> --cores <jobs/rows> instead: the rows share the
budget, and the +1 lets small derivations (the zip, a wrapper script)
run beside them.  --cores does not reach the output bytes: xz is pinned
to one thread, and the three tarballs came out byte-identical to a
serial build of the same commit.

plugins/Makefile built the seven Rust plugins with seven `cargo build
--bin X` rules.  With make -j they started at once and queued on
cargo's target-dir lock, so after the shared dependencies each final
crate and link ran alone.  One grouped rule now builds them all with a
single `cargo build --bin a --bin b ...`, prefixed with `+` so cargo
takes its job slots from make's jobserver rather than starting nproc
rustc processes beside make's own jobs.  A developer's `make` rebuilds
the same things as before: nothing on a no-op, all seven plugins when a
plugin source changes.

Measured on 24 threads, host nix, all three rows with their toolchains
already in the store:

  serial driver, one row at a time    10m46s
  one nix build, rows in parallel      5m37s

Running build threads (sampled every second across every nixbld
process, rustc codegen threads included) peaked at 26, and were above 24
for 6 of 294 seconds -- when a row's cargo and its C link steps meet.
The serial driver never got past an average of 3 busy CPUs.  Re-measured
after rebasing onto gcc 16.2.0 / rustc 1.98.1 / -O2: --rebuild of all
three rows took 6m00s, peak 26, above 24 for 5 of 324 seconds.
…ered plugins

The smoke test at the end of `all` took 83-88 s per row, run one row after
another: 4m19s for the three.  Nearly all of it was smoke.sh's
getmanifest handshake, which fed each plugin its request, held stdin open
with a fixed `sleep 3`, and did that for 27 plugins in turn -- 81 s of
sleeping per distro, and four times that per row in `accept`.

Measured, the wait is not needed: every plugin answers in under 0.25 s
natively and under 0.9 s under qemu-arm.  smoke.sh now starts all of them
at once, keeps each one's stdin open only until its reply arrives (polling
for "result", 20 s cap), and reports them afterwards in a stable order.
portability.sh runs its distro images concurrently -- each container only
reads the shared unpacked tree -- and prints them in the listed order; a
failed image now shows its FAIL lines first, since they come last in the
log and the 40-line head used to cut them off.  The driver runs
portability.sh for every row at once in `all` and `accept`, printing each
row's output whole.  The two-tree probe in `accept` stays serial: it
builds, and a build already uses every core it is given.

Timings, three rows, 24 threads:

  all, ubuntu:26.04 per row       259 s  ->   7 s
  accept, the full 11-image matrix ~17 min ->  11 s  (portability part)

Still caught, checked with a tarball carrying three bad plugins on two
images: one that never answers, one that answers with an error, and one
that exits at once.  All three are reported FAIL on both images and the
run fails, in 26 s.
The static musl lightningd was 16% slower than a glibc build, most of it
musl's malloc and memcpy under sqlite3_expanded_sql in db/db_sqlite3.c.
Replace the allocator in every shipped ELF -- the C binaries, the Rust
plugins (whose System allocator is malloc) and VLS -- by linking
nixpkgs' mimalloc.o on all three link paths in static-pie.nix.

The object rather than libmimalloc.a: an archive member is only pulled
in for an undefined symbol, so libc.a could satisfy malloc first and
the release would ship musl's allocator without any error.

mimalloc's version is now listed in each row's input manifest.

musl's x86_64 memcpy is byte copies to alignment, `rep movsq`, then a
byte-by-byte tail.  For the short copies sqlite3_expanded_sql() makes on
every database write it is about 5x slower than LLVM libc's, and with
mimalloc in place it was most of the remaining gap to a glibc build.

Build LLVM libc's memcpy.cpp and memmove.cpp (21.1.8, from nixpkgs' own
source) for baseline x86-64 and link them with mimalloc on every link
path.  memmove has to be replaced too: musl's calls __memcpy_fwd, which
would pull in musl's memcpy.o next to ours.  clang builds them because
the row's gcc produced a memmove 3-4x slower on large copies.  The arm
rows already get an optimised memcpy from musl and are unchanged.

3-node line benchmark, 3000 payments, 5 interleaved rounds (medians):

  build                       payments/s   CPU s, all nodes
  glibc, -O2 (same commit)    614          7.7
  musl, mimalloc              575          10.0
  musl, mimalloc, LLVM mem    597          8.3
…ce it

The armhf row used nixpkgs' armv7l-hf-multiplatform, which is Debian's
baseline: VFPv3-D16, no NEON.  The Rust plugins did not even hold to
that.  rustc ignores the row's gcc.fpu, and its armv7 musl target leaves
D16-D31 enabled, so cln-lsps-client, cln-lsps-service, clnrest and
wss-proxy died with SIGILL on a VFPv3-D16 CPU.  lightningd, cln-grpc
and the others also carried NEON code from OpenSSL's assembly.
check.sh only printed lightningd's attributes, so none of this failed
a build.

The arm tarballs are for the Raspberry Pi.  Every Pi that runs ARMv7
code (Pi 2 and later, Zero 2 W) has NEON and VFPv4 with 32 D
registers, so build armhf for armv7-a with neon-vfpv4: C through a
package set with that gcc.fpu (pkgsCrossRaspberryPi), and Rust with
`-C target-feature=+vfp4,+neon`, taken from the same fpu.  The fpu is
listed in the manifests.  amd64 and arm64 build recipes are unchanged.

check.sh now enforces the target instead of only printing it:
  * armhf: every ELF must be v7, at most VFPv4 and NEONv1, hard-float;
  * arm64 and armhf: every ELF is run under the pinned qemu on the
    oldest Pi's CPU (cortex-a53, cortex-a7) and must not die by
    signal.  This catches code the attributes do not declare, like the
    D16-D31 use above.
vls-check.sh runs VLS on the same CPUs.
The official release tarballs store usr/local/..., and the documented
install is `tar -xf ... -C /usr/local --strip-components=2`.  With the
leading ./ that command strips only . and usr, so everything lands in
/usr/local/local/ and the installed binaries are silently left alone.

Drop the ./ and make check.sh reject any member outside usr/local/.
The release manifests carried DIGESTS-<ver> and DIGESTS-<ver>-armhf, the
per-architecture and index digests of the published image, and three people
signed them. Nobody can check that signature: the image has one unpinned,
network-touching input -- the apt-get install of the entrypoint's runtime
dependencies -- so a third party rebuilding later gets different bytes and a
different digest.

The tree already said so in two places. verify's DIGESTS branch was explicitly
informational and never blocked, because "images are optional for a co-signer,
and the layer can drift between sittings". The Dockerfile calls the apt step
"the one unpinned, network-touching input". So the manifest required three
signatures over a hash we had twice written down as unverifiable. ("Bit-identical
across two --no-cache runs" was one machine at one moment, not a third party
later against a moved Debian package set.)

What is attested instead is stronger and one level down: the tarballs are built
from source under Nix, reproduce byte for byte, and carry three signatures -- and
the image is built from those signed bytes.

publish loses a guard with them, so it gets a better one. It used to refuse when
the layout's index digest was not the recorded one, which was really a staleness
check wearing an integrity costume. It now reads the layout's own OCI labels and
refuses when image.version is not this release or image.revision is not the
commit the manifest names -- the same question, answered by something that does
not have to be reproducible to be true.

The digests are still computed and logged for an operator comparing what Docker
Hub ended up with; they are simply not artifacts and not signed.
`SKOPEO_IMAGE=quay.io/skopeo/stable:v1.22.3` looked like a pin and was
not one. Quay's version tags float: the tag was re-pushed on 2026-10-02
and again on 2026-10-07. Worse, quay garbage-collects a manifest once no
tag points at it, so a digest taken from a floating tag does not merely
stop being current -- it stops existing. A digest pinned from v1.22.3
was verified working on 2026-09-30 and was `manifest unknown` by
2026-10-07, which broke the `docker` verb outright.

Now pinned by *index* digest of upstream's `v1.22.3-immutable` tag,
exactly as NIX_IMAGE is and for the same reason, with the version kept
beside it as SKOPEO_IMAGE_TAG so the diff stays readable. The immutable
tag has not moved since 2026-09-21, and because a tag references it the
manifest stays alive. The pins file's claim that every pin on the route
is "bumped deliberately, reviewed as a diff" now holds.

The comment also records why 1.22.3 and not 1.24.1, because the version
numbers invite exactly that question: v1.23.0, v1.24.0 and v1.24.1 exist
as *source* releases, but no 1.23.x or 1.24.x container is published --
not on quay.io/skopeo/stable (all 77 tags enumerated), not on
quay.io/containers/skopeo, not on ghcr.io, not on Docker Hub. skopeo
maintains several stable branches at once and backports to them; v1.22.3
shipped the same day as v1.24.1, and the image track follows one of
those branches. The note says not to "fix" the pin upward without
checking that an image exists first.

Verified: the same skopeo 1.22.3 (commit 92397d5b), a four-platform OCI
index, that it runs, and that it still has --preserve-digests -- which
is the reason the pinned container exists at all, Ubuntu 22.04's skopeo
1.4.1 lacking it. The `docker` verb's smoke test passes all three
platforms again: amd64, arm64 and arm/v7 each run `lightningd --version`
under binfmt.
The default was today's date, which made the release bytes a function of
when the build ran: two hosts on either side of UTC midnight, or a
co-signer a week late, would build byte-different tarballs from the same
commit.  Pin the baked timestamp to the committer date of the commit the
flake actually builds, which every build path now passes through
prepare_tree; verify keeps reading the manifest's own '# mtime' line and
--mtime still overrides.  The date-only shape is unchanged, which is what
tar's --mtime="$MTIME 00:00Z", configure's MTIME and docker's
SOURCE_DATE_EPOCH all consume.
armhf was scoped out of the required co-signed set while its Rust
binaries diverged across machines -- the rustc-1.89-era reordered ARM
EHABI string table -- and its artifacts lived in a captain-only
SHA256SUMS-<ver>-armhf.  That divergence is gone: with the mtime pinned
(now the commit-date default) and the current toolchain, two x86-64
hosts of different CPU vendors and an outside co-signer produce
byte-identical armhf tarballs.

So the split is ritual now, and ritual in the signing path is where you
do not want ritual: sign writes one manifest with every row and the
source zip, publish expects that one, and the @cosign-optional directive
and its reader are deleted.  verify keeps handling the legacy -armhf
manifests from split-era releases -- it reads whatever manifest it is
handed -- with their withhold-on-mismatch path intact, since that is
exactly how those releases were signed.  The docs' and release notes'
reproducibility caveat retires with it.
Rebasing onto v26.12rc1 brought in master's switch to nixpkgs'
lowdown-unsandboxed and the removal of the unconditional
`export PYTHON=python3` (configure now falls back to python3 itself).
Both touch nix/pkgs/default.nix, so its recipe hash moved.  The nixpkgs
pin and the toolchain list are unchanged; this is
`tools/reprobuild certify --no-verify`.
@cdecker

cdecker commented Nov 18, 2026

Copy link
Copy Markdown
Member Author

Rebased onto master (v26.12rc1), and a new snapshot to verify.

The branch conflicted with master in configure: niftynei's macOS fix (d8507fa) made the configurator's -I$CPATH/-L$LIBRARY_PATH conditional, while the cross/static commit here drops --configurator-cc from the same line. The resolution keeps both. Two other changes came in with the rebase:

  • master's switch to lowdown-unsandboxed and its removal of the unconditional export PYTHON=python3 both touch nix/pkgs/default.nix, so its recipe hash moved. The input manifests are re-certified in the last commit (8ec987d); the nixpkgs pin and the toolchain are unchanged, and a full certify (from-source build + acceptance suite) passed.
  • the four Actions workflows this PR deletes had only received action version bumps on master, so they stay deleted.

Head is now 8ec987d2c. The build, the Docker images and their smoke test pass on all three rows. Here is the signed manifest:

SHA256SUMS-v26.06-862-g8ec987d
# Core Lightning v26.06-862-g8ec987d
# rev 8ec987d2c5f6084714844ddfed856bc238a74ffa
# narHash sha256-x3t2ku99+/t7ln5HHjxI2awuQvsnd+S7fzPMlQwOO0o=
# submodules true
# mtime 2026-11-18
# nixpkgs-certified b4fd65b198c599cbe814fcb9f42d25d021595ec9
66b49c218f1ffd13d86ef489db39455505e31e9a09aac8277a22a6fcf4a9d2ee  clightning-v26.06-862-g8ec987d-static-amd64.tar.xz
8e6a7a02382762b65ef017498a7c0b2c0cb5c29c62528a8548da5cf6f2e9543f  clightning-v26.06-862-g8ec987d-static-arm64.tar.xz
540f481d51d9ab8cff6e21f1a4f27a00a14fa69127141007f3ac3654fccdd3d2  clightning-v26.06-862-g8ec987d-static-armhf.tar.xz
2dbfad4ab747109ce0cd292f10cf71c67bd25efa779044512f4d82f3f5966f19  clightning-v26.06-862-g8ec987d.zip
SHA256SUMS-v26.06-862-g8ec987d.asc
-----BEGIN PGP SIGNATURE-----

iQEzBAABCgAdFiEEt8S+gRhPwgPVLDXFFBbYPcTw6G0FAmr9sDkACgkQFBbYPcTw
6G0V9gf+NjnaFMW1ELN/3cx1R2dzVDsoZIcGIYDiVWRIsgn66ZWVCVbsyt/IUb6+
Zi495SNjNzJMYOQtI0qksq/irQz/enagHnVHgj1S+u8HVq2n1owS6Q8EPMh8WcQo
htZ8SECEhOWwr09ks/6xmA/6JMWgQJWkcIvqJxSpZnByfcD8cBLpSBfMZfKNiKKz
hwIxylyuF2D3nq2As4sItGLhenZg8cQ5xw/lJzCNx4xMklu2iFSVsvmk0fWVNIiD
vgBlmALavGYgqPj0dRMtjJLO27ud/oIVr1XFab7avTEiQndxM9Bbty3yMD11eRMA
CBzSB7H33ErF2q4qeB0tMImA6Fj34A==
=jUKv
-----END PGP SIGNATURE-----

To verify, save the manifest under exactly that name and let the driver rebuild it. It takes rev and mtime from the header, so you only need the commit in your checkout, not the branch checked out:

git fetch origin pull/9639/head
mkdir -p release   # save the manifest above as release/SHA256SUMS-v26.06-862-g8ec987d
tools/reprobuild verify release/SHA256SUMS-v26.06-862-g8ec987d

Add --no-sign for a dry run. Without it, a full match writes release/SHA256SUMS-v26.06-862-g8ec987d.asc.<keyid>; please post that here. Note that the rebase moved the commit date, so mtime is now 2026-11-18 and none of the hashes from earlier snapshots carry over.

@cdecker
cdecker marked this pull request as ready for review November 18, 2026 14:06

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants